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I . INTRODUCTION 



A. BACKGROUND 

A major problem for the U S Navy flight program is pilot 
workload saturation in the cockpit. This creates a 
situation in which there is no reserve to deal with 
unforeseen contingencies and no room to add other needed 
tasks. The Naval Air Development Center (NADC) located at 
Warminster, Pennsylvania, is particularly interested in 
finding a solution which can be implemented without creating 
a corresponding degradation in the performance of the 
individual aircraft's mission [Interview with Warner, 1987]. 

Research in the area of dual-task performance indicates 
four ways of tackling this problem [ Wickens, 1981 ] . 

1. Training pilots to perform multiple tasks more 
efficiently. 

2. The use of criteria which select those individuals 
with more proficient time-sharing skills in the 
recruitment of pilots. 

3. Designing the cockpits so that the pilot will only be 
requested to concurrently perform tasks which are more 
efficiently shared. 

4. Design the placement of controls and displays in the 
cockpit to allow for greatest time-sharing efficiency. 

The design of the cockpit has been studied extensively 
so that it is doubtful that any further large reductions in 
workload can be achieved in that area. Pilot selection 
could not be based solely on time-sharing abilities even if 
techniques for accurate measurement of this ability were 
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currently available. Pilot selection must be based on many 
more important criteria. Also, since pilots already receive 
extensive training it is doubtful that any significant 
increases in efficiency can be achieved by providing 
additional training. Therefore, attention is being focused 
on the third alternative of trying to ensure that tasks, 
which a pilot must perform concurrently, are compatible for 
the purpose of efficient time-sharing. 

Recent studies indicate that the use of voice 
recognition/synthesis in an aircraft cockpit may provide the 
best means of achieving the compatibility of tasks which 
will achieve the desired goal of workload reduction. 

B. LITERATURE SEARCH 

A large volume of material dealing with operator 
workload has been generated in recent years. This interest 
in operator workload is most likely the result of the 
existence of jobs which require multiple tasks to be 
performed in a very complex environment. An excellent 
example of such a job is the piloting of a U. S. Navy 
aircraft . 

Literature generated in the area of human performance 
covers all aspects of time-sharing, display design, single 
and multiple tasking, attention resources, etc. Because of 
this proliferation of research it is necessary to limit the 
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review of this material to that of attention resources and 
time-sharing. These two areas provide the theoretical basis 
for pursuing the use of voice recognition/synthesis in an 
aircraft cockpit as a means to reduce pilot workload. 

1 . Attention as a Limited Resource 

In 1967 Moray published a paper which highlighted 
the quantitative aspects of attention. He asserted that 
attention should be thought of as having a limited capacity 
in much the same way that a given computer is limited in how 
much data it can process. Additionally, he stated that this 
capacity could be divided into various amounts to be 
assigned to tasks on the basis of difficulty and demand. 
Particularly beneficial was the flexibility and ability to 
share these resources that was highlighted in his treatment 
of the subject. [Wickens, 1984 :pp. 292-293] 

2 . Kahneman Model 

The Kahneman model is important because it was the 
first attempt to predict performance based on the limited 
capacity/resource theory. However, his model still dealt 
with attention as having a single source capacity and could 
not account for instances in which more difficult secondary 
tasks interfered less with the primary task than other less 
difficult tasks. This seemed to indicate that the theory of 
a single resource was possibly too simplistic. 

[ Wickens , 1984 and Wickens, 1987 ] 
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3 . Multiple-Resource Theory 

The single resource theory of the Kahneman Model was 
valuable in asserting the idea of attention as a limited 
capacity resource. However, it failed to explain 
differences from predicted results which occurred when 
certain tasks were combined. [Wickens, 1984] 

In an attempt to resolve these apparent 
discrepancies, Wickens developed a Multiple-Resource model 
which combined not only the theory of attention as a 
resource but also the theories of attention as structural in 
nature [Wickens, 1981, Wickens, undated , Wickens, 1988 and 
Wickens, 1984 ] . 

Building on research by individuals such as Pachella 
(1974), Kinsbourne and Hicks (1978), Harris, Owens and North 
(1978) and others ( Wickens , undated ] ; Wickens designed a 
three dimensional model (Figure 1) which describes attention 
along the dimensions of (1) stages of processing, (2) 
cerebral hemisphere of operation and (3) modalities of 
processing (encoding and response). 

Stages of processing include the processes of 
encoding, memory & transformations, and responding. The 
idea of different sequentially ordered stages of processing 
has achieved a certain level of acceptance and even though 
it cannot be inferred that independent resources are used. 
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some research in dual tasking seems to indicate it could be. 
[Wickens, undated] 

STAGES OF PROCESSING 




(Adapted from Wickens (undated)) 

The research done in the area of separate cerebral 
hemispheres of operation argues strongly that the two 
cerebral hemispheres act as separate resource reservoirs. 

In particular, spatial tasks along with left hand controls 
are handled by the right hemisphere of the brain whereas 
verbal tasks and right hand controls are handled in the 
left. Since a slight additional amount of time is needed in 
order for the brain to send messages between the hemispheres 
of the brain, it is considered advantageous in certain 
circumstances to keep processing of a single task confined 
to one hemisphere. However, in other situations it is 
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beneficial to use both hemispheres since competition for the 
same resources can cause a slower response than that 
resulting from hemisphere crossover time. 

Some of the clearest measurements have been along 
the dimension of different modalities of processing in which 
auditory versus visual encoding, and manual versus vocal 
responses have been studied. Sandry demonstrated the effect 
of these modalities in her experiment conducted on the F-18 
simulator located at the Naval Air Test Center, Patuxent 
River, Maryland in 1981 [ Sandry, 1982 ] . 

4 . Sandry Study 

In her testing, Sandry paired verbal and spatial 
tasks* with the continuous primary spatial task of flying 
an aircraft. By comparing reaction times of different 
input/output modality pairs, inferences concerning expected 
performance could be made. In general, she found that the 
time required to complete a verbal task could be reduced by 
changing to a modality combination which utilized voice 
recognition and/or synthesis while such a combination 
resulted in degraded spatial task performance. 



*Wickens defines verbal tasks as "Tasks for which words, 
language, or logical operations are natural mediators ...". 
Spatial tasks are defined as "... almost any task that requires 
moving, positioning, or orienting objects in space, or performing 
other analog transf ormations, . . . " [ Wickens, 1987 : p. 16]. 
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sandry's study was useful In two ways. First, she 
directs attention immediately to those tasks which will be 
most benefited by voice implementation. Second, because the 
data was gathered on the Navy's F-18 simulator, it can be 
used for an initial cut in projecting possible reaction time 
savings . 

C. DEPARTMENT OF DEFENSE STUDIES 

1. Air Force B-52 Bomber Study [North and Lea, 1982] 

This study represents the armed forces' first 

attempt to identify cockpit tasks which would be benefited 
most by conversion from visual/manual input/output 
modalities to audio/vocal modalities. The approach 
developed by Honeywell dealt with cockpit tasks on an 
individual mission basis. A filtering system was then 
developed which screened and prioritized candidate tasks 
using a series of objective and subjective criteria. 

2. Navy F/A-18 Study [Mountford, North, Metz, and 

Graf funder , 1982 ] 

The Navy study, which was also conducted by 
Honeywell, had three objectives. They were to (1) perform a 
literature review, (2) develop task selection procedures, 
and (3) study 'dialogue' issues. Of primary interest to 
this thesis is the second objective of task selection. The 
resulting procedures refined those developed as part of the 
B-52 study. 
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These two studies performed by Honeywell, although 
valuable, are weak in two areas. First, determination of 
technological feasibility for a particular candidate task 
for voice implementation is done with a weighting factor. 

It would be much more straight forward to use a go/no-go 
decision process for this. When Warner at NADC attempted to 
use the proposed filtering process, tasks which were not 
technologically feasible were not successfully screened out 
[Interview With Warner , 1987 ] . 

Secondly, the filtering process designed during 
these studies chooses and prioritizes candidate tasks on an 
individual basis with only a limited ability to judge the 
interference effect when multiple tasks are converted to 
voice. There is no attempt to determine the most 
advantageous combination of voice tasks. 

D. DEPARTMENT OF DEFENSE FLIGHT TESTS 

The military services, in spite of some valid 
reservations, have displayed a continued interest in voice 
synthesis/recognition as a possible solution for the 
saturated workload situation present in the aircraft 
cockpit. As a result, flight tests have been performed, the 
results of which are discussed below. 
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1. F/A-18A Flight Tests [Loikith and Hall, 1986] 



The purpose of these flight tests was to determine 
the feasibility and pilot utility of voice 

synthesis/recognition. Results of the tests indicated that 
voice implementation of cockpit tasks was feasible and could 
be useful provided that technical problems with the 
particular voice system tested could be corrected. The Lear 
Siegler, Inc; Voice Control Interactive Device was used for 
these tests. 

2. AFTI/F-16 Flight Tests I Williamson, undated ] 

Using the Texas Instrument Voice Interactive System 
the Air Force obtained impressive results. The following 
excerpt from the "Discussion" portion of the report reveals 
how well this testing went. 

The test pilots found it safer and more convenient to use 
voice instead of traversing the awkward menu logic of 
multifunction keyboards and displays. The ability to 
request information verbally and receive a verbal response 
has also received universal praise from the pilots. 

This report also highlights the fact that 
development of a method for selecting the most promising 
tasks for conversion to voice is necessary. 

3. JOH-58C Helicopter Flight Tests [ Szerszynsk i , 19 87 ] 

The Army also experienced favorable results using 

voice in the JOH-58C helicopter. One of the sub-tests 
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conducted was a simulated single pilot missior/ during 
which the pilot had to simultaneously control flight tasks, 
radio control, and visual search . Their assessment of the 
test results was that "The voice controlled avionics system 
is, based on the data, the avionics control system of choice 
when the workload is high". 

These tests all demonstrate the ability of voice 
synthesis/recognition as a means to reduce pilot workload 
thereby increasing efficiency and safety. The last two 
tests in particular confirm the fact that the necessary 
voice technology is currently in existence. 
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I I . MODEL 



A. OVERVIEW 

The primary goal of this thesis is to combine previous 
research and DOD studies into a single model which will 
provide the design engineer with guidelines and procedures 
for a total systems approach to implementing voice 
recognition/synthesis in the cockpit. Wickens states the 
goal of this model best. 

A major goal of predictive performance models is to help 
determine the most effective design before a system is 
configured. Such models are not likely to be 100% 
reliable. However, they may prevent the designer from 
developing prototypes that will be clearly non-optimal, 
and will also enable the designer to identify a parameter 
space that is most worthy of experimental investigation. 

[ Wickens, 19883 

Likewise, this model will provide the most beneficial 
starting place for design and testing but cannot be expected 
to provide "the answer". Variances between aircraft and 
ever changing missions preclude any other approach. 

This model draws extensively from the methods developed 
in the Honeywell studies [North and Lea, 1982 and Mountford, 
North, Metz and Graf funder , 1982 ] which use a total systems 
approach in deciding what systems to activate by voice. The 
performance data generated by Sandry (1982) provides the 
data for use in projecting expected workload savings. The 
product of this model will allow the design engineer to 
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begin with a first cut will maximize workload savings while 
avoiding overloading the pilot on the voice interactive 
side. 

To aid in the future refinement of this model, 
individual modules will be used to separate distinct 
portions. Detailed descriptions of these modules follow. 

B. MODEL DESCRIPTION 
1 . Module One 

a. Task Analysis: 

This portion of the model was developed by 
Honeywell [North and Lea,1982:pp. 8-14 and Mountford, North, 
Metz and Graf funder , 1982 : pp. 6-18] as part of two studies 
performed for NADC. 

(1) Mission Scenario : According to the 

Honeywell studies, the first step in analyzing an aircraft 
cockpit regarding the value of implementing voice 
synthesis/recognition is to develop a mission scenario. 

This scenario should be a general narrative which accurately 
describes the mission under consideration. 

Honeywell deals with mission scenarios 
individually, taking each mission separately and running it 
through the entire procedure. In this model a broader look 
is taken. All of the major missions of the particular 
aircraft under study should be developed and evaluated 
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together to provide an analysis which produces 
reconunendat ions for voice implementation based on the 
aircraft as a total system. 

In choosing the missions to evaluate, 
consideration should be given to high workload missions 
although it is important to include all major missions. 
Infrequently flown missions should not be considered unless 
unusually critical since they will unduly affect the 
findings of the evaluation process. 

(2) Task Narrative : The second step advocated 

by Honeywell, is the development of a task narrative which 
provides a detailed verbal description of each task of the 
mission. A task narrative should be developed for each 
mission scenario. An example of a task narrative is 
contained in the F-18 Honeywell study [North and Lea,1982:p. 
9] and has been provided as Table 1. The tasks should be 
listed in the order in which they occur during the flight. 

(3) Time-based Activity Log : The next step in 

the Honeywell procedure is to use the Task Narrative to 
"...categorize each activity into its human information 
processing channels--vocal, visual, auditory, and manual." 
[Mountford, North, Metz, and Graf funder , 1982 : p . 10]. Also 
an index should be included which indicates the difficulty 
in distance and location the pilot experiences in reaching 
for the manual control. Those tasks for which the pilot 
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does not reach would be assigned a value of zero and 
increase in incremental steps of one up to the most 
difficult. This reach index is a useful measure when 
combined with other information in deciding the best 
candidates for speech synthesis/recognition. In Honeywell's 
F-18 study example the highest value was a three. Figure 2 
provides an example of how this index could be assigned to 
an aircraft cockpit. 



Table 1. TASK NARRATIVE 
(Adapted From North and Lea (1982)) 



Pilot Tasks 


Copilot Tasks 

L .. 


Additional Aircrew Tasks 


Calls for preparation for 
contact checklist. 


Reads preparation for 
contact checklist. 


RN — Calls range to tanker in 
1 NM increments. Calls 
range at 3 ... 2 Mi. 


1 . Establish radio contact 


2. Check air conditioning 
system. 








N— 3. Set rendezvous radar 
to standby. 




A. Torn anticollision lights 
off. 






5. Turn naviqat ion 
lights to flash. 




6. Di sengage autopilot . 


6. Set slipway and airplane 
lioht switches to full 
bright . 




7. Set airbrake lever 
to position 1 . 


7. Open slipway doors and 
verify ready lights on. 






8. Determine tanker positJon 
on FLIR sensor. 


RN — 8. Determine tanker 
position on FLIR 
sensor. 


9. Select FLIR video. 


10. Set air refueling 
switch to air refuel. 
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Figure 2. Anthropometric Zones 
(Adapted From North and Lea(1982)) 
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An example of the Time-based Activity Log is 
included as Table 2 [Mountford, North, Metz, and 
Graf funder , 1982 :p . 111. Immediately obvious in this sample 
is the fact that many of the tasks show input or output 
modes of "N" (not applicable). This is not appropriate. 

Each task can have both input and output modes specifically 
identified. For the case where a "not applicable" entry 
appears appropriate the following guidelines should be 
employed. If the task is prompted by a checklist it should 
be considered to have a visual input mode. If it is a 
memory item, then it should be listed as an audio input mode 
because this most closely identifies the information 
processing channel (hemisphere) used. In the case of output 
modes the vocal mode should be identified. This follows 
from the fact that if no obvious manual action occurs then 
the information processing channels activated must be the 
same as those used for a vocal response. 

(4) Activity Chart : This chart is the graphic 

form of time-based activity log. See Figure 3 below 
(Mountford, North, Metz, and Graf funder , 19 82 : p . 13). These 
charts are particularly important to the subsequent 
screening of proposed tasks for voice implementation since 
they provide quick information on which tasks are performed 
in conjunction with the flight task thereby creating a dual 
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Table 2. TIME-BASED ACTIVITY LOG 
(Adapted from Mountford, North, Metz, and Graf funder (1982)) 
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tasking situation. This information will be necessary 
during the optimization portion of this model. 



TAXEOM 

m.0T coniirr 

OtSCAETE COKTIKUOUl OIICACTE C3KTMU0US 




Figure 3. Activity Chart 

(Adapted From Mountford, North, Metz, and Graffunder (1982) 

b. Development of Speech I/O Candidate Task List: 
After producing the task analysis for all 
pertinent scenarios, an initial list of potential 
candidates for conversion to voice should be compiled. 

Each task should be evaluated by the following series of 
filters. These filters provide the first gross look at a 
given task to judge its suitability for voice. 
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(1) verbal Task Filter: This first filter asks 



whether or not the task can be classified as one using the 
verbal information processing channels (see Footnote 1). If 
the answer is yes, it will continue through the filters 
which follow. If no, it will be rejected immediately. 
Currently, only verbal tasks benefit from conversion to 
voice synthesis/recognition. 

(2) Speech Recognition Filter : This filter is 

used to decide whether a task would be a good candidate for 
speech recognition. The questions for this filter were 
developed in the Honeywell B-52 study, further refined in 
the Honeywell F-18 study and are provided in flowchart form 
as Figure 4 [Mountford, North, Metz, and Graf funder , 1982 : p . 
20 ] . 

One additional question has been added to 
this filter. The first question asks if the task is 
currently accomplished with a by pilot speech. If the 
answer is yes, it will be immediately rejected since it 
already uses voice output. If no, it will continue through 
this filter. 

The question concerning the number of times 
the task is performed during the mission has been 
transferred to its own filter so that this it can be more 
efficiently handled. 
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Figure 4. Speech Recognition Filter 
(Adapted From Mountford, North, Metz, and 
Graffunder (1982) and Modified By Author) 



(3) Speech Generation ( Synthesis ) Filter : This 

filter is used to decide whether a task would be a good 
candidate for speech synthesis. A flowchart for the 
following questions is provided in the Honeywell B-52 study 
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[North and Lea,1982:p. 21] . It should be noted that 
question one was added to the filter by this author. 

1. is the task currently handled by speech input to the 
pilot? If yes, reject task. If no, continue to 
evaluate . 

2. Can information be transmitted to pilot/co-pilot as a 
short phrase? If no, reject task. If yes, continue 
to evaluate. 

3. Does it occur during visual time-sharing? If yes, 
accept task. If no, continue to evaluate. 

4. Can it be used as an information display that 
currently does not occur in the central anthropometric 
visual zone? If yes, accept task. If no, continue to 
evaluate . 

5. Can it be used together with speech recognition in an 
interactive dialogue format? If yes, accept. If no, 
reject . 

(4) Technical Feasibility : This filter is used 

to decide whether a task can be implemented as part of the 
voice system from a technical standpoint (e.g., if a task is 
using equipment not controlled through the multi-plexer 
(mux), the voice system cannot access it and the task will 
be dropped from the candidate task list). This is a 
question which must have a yes or no answer from an 
engineering standpoint. Therefore, each task which was 
accepted by either the Speech Recognition Filter and/or the 
Speech Generation Filter should be evaluated technically. 

This procedure differs from the Honeywell 
studies which address this issue from the standpoint of a 
feasibility scale which is judged to provide little 
additional information and only serves to obscure the 
analysis by adding an additional factor. In fact, the use 
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of such a scale could lead to accepting a task which cannot 
be implemented. 

(5) Repetition Filter : At this point, the tasks 

which have successfully passed through the previous filters 
should be grouped according to equipment used, input/output 
mode, number of steps to complete, and anthropometry index. 

This is done so that the frequency with 
which the same actions are taken can be judged*. Once this 
grouping has been compiled, the decision maker must decide 
how many times a task must be repeated before it becomes 
advantageous to consider conversion to voice modes. 

Once this decision has been made, only those 
tasks which occur with the proper frequency will be placed 
on the Candidate Task List. 

(6) Candidate Task List : Once a task has been 

accepted as a candidate for voice synthesis and/or 
recognition it should be placed on the Candidate Task List 
which is the output for this module. Along with each task 
the current input/output modes and the anthropometry index 
should be listed. An example of the procedure described in 
this module is included in the Appendix. 



^This handling of task repetition differs from the Honeywell 
studies in that instead of being part of the Speech Recognition 
Filter, it is handled as a separate filter. The reason for this 
change is that grouping the tasks as described above prior to 
asking this question simplifies the process considerably when 
this procedure is put into practice. 
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2 . Module TWO--P1 lot Input 



Pilot involvement in the task selection process is 
considered essential by NADC [Interview With Warner, 1987). 
Pilots possess a wealth of knowledge and experience, which 
if properly used, will greatly enhance this selection 
process. In addition, the involvement of the pilots should 
result in greater acceptance of this new technology. 

Both of the Honeywell Studies [North and Lea, 1982 
and Mountford, North, Metz, and Graffunder, 1982] address 
the assignment of a pilot utility rating to those tasks 
which are contained in the Candidate Task List produced in 
module one above. 

In the Honeywell F/A-18 study [Mountford, North, 
Metz, and Graffunder, 1982], it was suggested that pilots be 
given a questionnaire which contains questions requiring 
checks in blocks which are scaled on an ordinal basis from 
one (never useful) to six (always very useful). The 
following example is taken directly from that study. 

Example: "Descent Checklist" 

1 2 3 4 5 6 

Never Always very 

Useful Useful 

Sometimes 

Useful 

Responses should be compiled to show both how many 
pilots selected a given rating and the average rating. This 
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allows the decision maker to choose either the average or 
that rating that was picked most often (the mode) as the 
combined utility rating to be used in the optimization 
module. The following example illustrates this process. 

Example: "Descent Checklist" 

1 2 3 4 5 6 

Never Always very Average 

Useful 1 4 8 15* 12 Useful 4.83 

Sometimes 

Useful 

indicates mode 

In this example, there is very little difference 
between the average and the mode even though these two 
figures could vary greatly. 

In evaluating the feedback provided by this 
questionnaire, strong consideration should be given to 
deleting any task from the candidate list which receives 
ratings which fall primarily in the 1 and 2 range. If not 
completely eliminated, these tasks could still be selected 
in the optimization module if other weighting factors are 
high enough. In most cases it would be counter productive 
to provide a voice task which the pilots have already 
indicated would not be useful. 

The output of this module is a list of candidate 
tasks along with the assigned pilot utility rating for each. 
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3 . Module Three--Per formance Specifications 

The goal of this portion of the model is to 
establish performance specifications which can be used in 
Module Four to maximize workload reduction (reaction time 
savings ) . 

These performance specifications must include both 
the single and dual task conditions. The reason for 
inclusion of the single task condition is that the mission's 
activity chart produced in module one includes time frames 
in which the pilot/co-pilot is not engaged with the flight 
task, thus becoming a single task situation. 

In establishing performance specifications, it is 
first necessary to determine whether the numbers should 
reflect the worst case, best case or expected (average) 
case. This must be decided on an individual case basis. 

One method for establishing these specifications is 
demonstrated using the data from the Sandry study 
[ Sandry, 1982 ] . As previously mentioned, reaction times and 
root mean squared error rates were gathered through use of 
the Navy's F-18 simulator. The pilots used were from the 
local flying club and had an average of nine years 
experience* . 



Since F-18 pilots are used for data gathering efforts only 
on advanced high level projects, the experience level of the 
pilots used by Sandry was appropriate. 
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For the purposes of this model, workload is defined 
by reaction time. This means that the reduction of the 
amount of time necessary to complete a task directly 
translates into a reduction of workload (reaction time 
savings) for the pilot/co-pilot. 

Prior to using the data collected by Sandry, its 
validity for the purpose of establishing performance 
specifications was evaluated. A linear regression model was 
used for this analysis. Using the Statgraphics Package by 
STSC, regressions were run for single verbal task reaction 
times, single spatial task reaction times, dual verbal task 
reaction times, dual spatial task reaction times, dual 
verbal task root mean square values, and dual spatial task 
root mean square values. The results were as follows: 

1. Residuals indicated that the linear regression model 
was appropriate for the reaction times but not the 
best choice for root mean square error rates (Figure 
5). However, since it is only necessary to judge 
general trends for the error rate (which were 
adequately identified by the linear regression), no 
additional models were evaluated. 

2. Verbal tasks but not spatial tasks were benefited by 
conversion to voice input and/or output modes. 
Therefore, no performance specifications will be 
developed for spatial tasks. 

3. Error rate declines when voice input and/or output 
modes are used (Figure 6). 

4. In a single task condition, a variance is found among 
the subjects, which although is statistically 
significant, disappears in the dual task condition. 

5. The independence of the variables was confirmed. 

In establishing the performance specifications for 
this demonstration, mean reaction times were used. 
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Cumulative Percent 



SINGLE TASK 

NORMAL PROBABILITY PLOT 



DUAL TASK 

NORMAL PROBABILITY PLOT 




Residuals 



ROOT MEAN SQUARE 
NORMAL PROBABILITY PLOT 




-7 13 33 53 73 

Residuals 



Figure 5. 



Normal Probability Plots tin secs] 
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Component Effect 



COMPONENT+RESIDUAL PLOT 
SINGLE TASK REACTION TIME 




COMPONENT+RESIDUAL PLOT 
DUAL TASK REACTION TIME 




Figure 6. Component +Res idua 1 Plots tin secs] 
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Single and Dual Task Input/Output Modes 



Figure 7. Plot of Reaction Time Variance^ 



The letters S and D were used to distinguish between single 
and dual tasks, respectively. The numbers shown with these 
letters identify the input/output modes as follows: 

1 = Visual/Manual 

2 = Visual/Vocal 

3 = Audio/Manual 

4 = Audio/Vocal 
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Figure 7 shows the spread of the reaction times by 
input/output mode and by single/dual task condition. The 
trend present in the intervals shown is consistent with that 
shown by the means. Therefore, the means can be used for 
ease of computation purposes without distorting the outcome. 
The increase in the variance between the single and dual 
task conditions is explained by the resource capacity theory 
discussed in Chapter I. The variance present, while a 
consideration, does not prohibit the use of the average in 
establishing performance standards. Table 3 below provides 
the performance speci f icat ions expected for tasks in each of 
the four combinations of input/output modes. 



Table 3. AVERAGE REACTION TIMES 



Single Task 
Reaction Times 



Dual Task 
Reaction Times 



Input/Output Mode 



Visual/Manual (V/M) 
Visual/Vocal (V/V) 
Audio/Manual (A/M) 
Audio/ Vocal (A/V) 



10.03 [secs] 



8.37 

8.57 

6.8 



15.69 

15.24 

11.59 



17.91 [ secs ] 
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Tables 4 and 5 provide the expected difference in 
reaction time which will occur if the input/output modality 
is changed. 

Table 4. SINGLE TASK REACTION TIME SAVINGS (seconds) 



To 





V/M 


V/V 


A/M 


A/V 


V/M 


0 


1.66 


1.46 


3.23 


V/V 


-1.66 


0 


-0.20 


1.57 


A/M 


-1.46 


0.20 


0 


1.77 


A/V 


-3.23 


-1.57 


-1.77 


0 



Table 5. DUAL TASK REACTION TIME SAVINGS (seconds) 



V/M 


V/M 

0 


V/V 

2.22 


To 

A/M 

3.67 


A/V 

6.32 


V/V 


-2.22 


0 


0.45 


4.10 


A/M 


-3.67 


-0.45 


0 


6.32 


A/V 


-6.32 


-4 . 10 


-6.32 


0 



The potential for reducing reaction time is found by 
locating the row which lists the input/output mode 
combination currently in use and combining this with the 
column indicating the candidate input/output mode. 

In many cases it will not be possible to pull the 
numbers from these tables and immediately use them. It is 
first necessary to relate the task at hand to those 
performed in the Sandry study. The most straight forward 
way to accomplish this is to consider the number of steps 
necessary to perform the task in question. The Sandry tasks 
required four steps for completion. Therefore, the reaction 
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time for the task under consideration can be obtained by 
multiplying the tabled reaction time by the ratio calculated 
by dividing the number of steps necessary to complete the 
current task by four. 

Tables 3, 4, and 5 represent the type of output 
expected from this module. 

4 . Module 4 — Optimization 

This portion of the model represents the heart of 
the total systems approach to implementation of voice 
synthesis/recognition in the cockpit. The goal of this 
module is to select that combination of candidate tasks with 
associated input/output modes which will provide the largest 
reduction in workload without overloading the voice channels 
used by the voice input/output modes. 

Linear programming is proposed as the preferred 

method for accompl ishing this objective. This method was 

developed as a means of dealing with situations containing a 

large number of variables with associated constraints. As 

stated by the Operations Analysis Group (1984), 

Problems of allocation arise whenever there are a number 
of activities to be performed but limitations on either 
the amount of resources or the manner in which they may be 
allocated prevents accompl ishment of each separate 
activity in the most effective way conce i vable . . . a 
powerful technique that has been developed to solve such 
problems is called mathematical programming. When the 
problem can be formulated within a mathematical framework 
in such a way that it becomes one of maximizing or 
minimizing a linear expression subject to certain linear 
constraints, the technique is known as linear programming. 
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Likewise, this is an allocation problem and it can 
be formulated within the "mathematical framework" of a 
linear function with linear constraints. Such a 



mathematical formulation follows. 



Maximize^ Lj E k Ej Tj^j 


X ijkl 


(2-1) 


such that 






Ej Ej E k E L r ijkl 


<= Cj 


(2-2) 


^k x ijll 


i— 1 

II 

V 


(2-3) 


J* 

It 

X 

K> 

II 


• 

m 

• 

II 

o» 


(2-4) 



where 



1 

j 

k 

1 



A i jkl 

X ijki 

Tjkl 



'3 

n 



= Candidate Task 
= Mission 

= Input/Output Mode 

= Type of Task (single or dual condition) 
= Utility Value ® Of Task i. Mission j, 

Input/Output Mode k, and Type of Task 1 
= The Variable of Choice 
= Reaction Time' For Task i. Mission j, 
Input/Output Mode k, and Type of Task 1 
= Total Time (in seconds) of Additional 
Time Allowed For Voice Modes 
= Number of Missions Under Consideration 



The objective equation in (2-1) maximizes the total reaction 
time savings for all of the missions under consideration. 

The constraint equation (2-2) restricts the amount of time 
which can be added to the voice information processing 
channels during a given mission so that the pilot will not 



*The utility value is the weighted combination of reaction 
time savings, pilot utility and anthropometry index. The 
importance of each element is dependent upon the judgement of the 
decision maker. One possible weighting scheme is demonstrated in 
the Appendix. 

7 

This figure is obtained from Tables 4 and 5 as appropriate. 
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become overloaded in that area. This value is at the 
discretion of the decision maker. Constraint equation (2-3) 
limits the selection of the input/output mode to only one 
for each task. Lastly, constraint equation (2-4) insures 
that the same combination of tasks will be selected for each 
mission. This constraint is deleted when the missions do 
not contain identical tasks. When this occurs it will be 
necessary to evaluate the missions separately to establish 
the proper mix of tasks. However, in general, the same 
tasks will be present in every mission. 

Any good linear programming package will be able to 
solve the above maximization problem, thereby supplying the 
system designer with the best initial combination of tasks. 
This combination of tasks will represent the largest 
workload savings possible while not overloading the pilots 
voice interactive channels. The LINDO (Linear, INteractive, 
Discrete Optimizer) software package by LINDO, Inc is used 
in the Appendix to solve an example problem. 

C. MODEL SUMMARY 

This model has intentionally been designed to be general 
in nature so that it is easily applied to any aircraft 
cockpit. The procedures described are specific enough to 
guide the system designer while retaining the flexibility 
necessary to address the needs of the individual aircraft. 
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In summary, this model consists of a series of questions 
which evaluate the potential of all tasks performed during 
all of the aircraft's missions. Those tasks which are 
considered as good candidates for conversion to voice 
synthesis/recognition are then rated by the pilots for 
usefulness resulting in a pilot utility index for each. The 
tasks are then evaluated by an optimization program using 
the performance specifications contained in Module Three to 
determine what combination of tasks will provide the 
greatest reduction in pilot workload without overloading the 
voice channels. 
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III. FUTURE STUDY 



A. FURTHER DEVELOPMENT OF MODEL'S INDIVIDUAL MODULES 

As stated in the overview of the model, individual 
modules representing distinct portions of the process have 
been used in order to facilitate the refinement process for 
this model. 

Module One is fairly well complete but the processing of 
this portion would be greatly enhanced through use of 
automation. A computer program (possibly in Fortran) could 
be designed which would ask the appropriate filter questions 
and then generate the Candidate Task List as the final 
output . 

Module Two requires pilot input and could also be 
greatly enhanced through the use of a computer program. The 
survey form could be automatically generated from the 
Candidate Task List prepared as part of module one. 

Module Three could be improved in several ways. 

Although useful, better data on reaction times could be 
collected for use in establishing performance specifications 
by using actual F-18 pilots. Even if F-18 pilots could not 
be obtained, additional tasks could be tested on the 
simulator in hopes of obtaining more accurate reaction 
times. In the mean time, possibilities for a more reliable 
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method of converting the values In Tables 3, 4, and 5 could 
be explored. 

Module Four will work well with any good optimization 
package but more study into different weighting schemes 
could prove to be valuable. Both modules three and four 
would benefit from being automated in a complete package 
with Modules One and Two. 

B. TRAINING IMPLICATIONS 

The training issue should be addressed for two reasons. 
Firstly, the Sandry study (1982) showed a significant 
reduction in reaction time for verbal tasks which were 
converted to voice synthesis/recognition input/output modes 
after only a small amount of training and practice. More 
dramatic results could occur with additional practice by the 
pilots. While pilots are practiced to the point of an 
automated response for visual and manual actions, this is 
not true for audio and vocal ones. Further, testing in this 
area could produce very favorable results which would lead 
to a greater recognition of voice interaction in the cockpit 
as a valuable asset for pilots. 

Secondly, methods for instructing pilots in the use of a 
voice interactive system should be explored so as to 
facilitate the training process. 
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C. SYNTAX DEVELOPMENT 

Selection of the words (syntax) to be used with the 
voice interactive system must be made carefully. These 
words must be meaningful to the pilot as well as 
distinguishable from each other by the recognition system. 
Words which sound too much alike would lead to a higher 
recognition error rate. 

Pilot input into the selection process would not only 
result in a superior vocabulary but would lead to greater 
pilot acceptance. 

D. SPATIAL TASKS 

At this time only verbal tasks have been considered as 
candidates for voice synthesis/recognition. Literature in 
the field as well as the Sandry study (1982) have clearly 
demonstrated the fact that tasks processed by those 
information processing channels which are utilized by 
spacial tasks do not benefit by conversion to voice 
input/output modes. 

However, the possibility does exist that some spatial 
tasks could be converted so that the verbal information 
processing channels would be used to perform the task. If 
this could be accomplished then conversion to voice modes 
could prove beneficial. 
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For example, a target tracking task could be designed 
such that the pilot could move a cursor to a designated grid 
by stating a letter-number combination rather than using a 
joystick to move the cursor to the same area. 

Creative designing of cockpit tasks could greatly 
enhance the use of the voice interactive system. 
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APPENDIX 



EXAMPLE OF MODEL PROCEDURE 

The purpose of this example is to demonstrate step by 
step the procedures described in the model contained in 
Chapter II. The treatment of the included tasks has 
deviated from fact in some instances so that various aspects 
of the model could be more fully illustrated. The 
information for this example has been obtained from the Task 
Narratives contained in Appendix A of the Honeywell B-52 
Bomber Study [North and Lea, 1982]. In this example the 
pilot's job will be analyzed. 

A. MODULE ONE 

Since the Mission Scenario and the Task Narrative are 
self-explanatory, this example will begin with the Time- 
based Activity Logs taken directly from the Honeywell study 
and contained here as Tables 6, 7, 8, and 9. As indicated 
by the logs, the mission segments are Takeoff, Lowlevel, Air 
Refueling, and Recovery. These segments are organized into 
two missions. Mission one consists of Takeoff, Air 
Refueling, and Recovery while mission two consists of 
Takeoff, Lowlevel, and Recovery. 
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Table 6. TAKEOFF ACTIVITY LOG 
(Adapted From North and Lea (1982)) 
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TURNS BOMB REL LUES 



Table 8. air refuel activity log 
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Activity charts have not been prepared for this example 
but it should be noted that single and dual tasking has been 
considered during this procedure as appropriate. 

The next step in the process is to screen each task 
listed on the Time-based Activity Logs for inclusion on the 
Candidate Task List. It was decided that a spreadsheet 
approach would best suit the procedure so Lotus 1-2-3 was 
chosen for this purpose. Tables 10 and 11 contain the 
spreadsheets prepared in the process of screening tasks 
included in missions one and two respectively. A step by 
step explanation of the process follows. 

The first step was to determine if the task was 
0 

verbal 0 . If the answer was affirmative, "YES" was placed 
in the appropriate column. If the answer was negative, the 
task was dropped from further consideration and "Reject" was 
entered. Once a task was rejected three dashes were placed 
in the remaining columns to indicate that this task was no 
longer under consideration for voice implementation. 

Next, each task was processed through the Speech 
Recognition Filter. In response to the question of whether 
the output mode was currently vocal, either "NO" or "Reject" 
was entered. If a "NO" response was entered the question of 
whether the task was manual discrete was asked. The entries 



0 

This term is defined in footnote 1. 
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Table 10. MODULE ONE SPREADSHEET- -MI SSI ON ONE 
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Table 11. MODULE ONE SPREADSHEET — MISSION TWO 
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were either "YES" or "Reject". If the answer was yes then 
the question was whether the task occurred during time- 
sharing. The entries were then either "Accept" or "NO". If 
no, then the task continued through the filter. Next, the 
question was asked about whether the Anthropometry Index was 
greater than zero. The entries were either "Accept" or 
"NO". For "NO" answers, the last question about whether the 
task resulted in information retrieval was asked. The 
entries were either "Accept" or "Reject". 

In this example, no questions were necessary after the 
time-sharing question although all of the columns were 
included for completeness. Lastly, all of the accepted 
tasks were indicated as such by an "X" placed in the 
appropriate column following the Speech Recognition Filter. 

Next, the Speech Generation Filter was applied. The 
questions in this filter were processed much the same as for 
the Speech Recognition Filter and should thus be self- 
explanatory. 

The last question included on the spreadsheet dealt with 
the technical feasibility of the tasks accepted by the 
previous filters. If the task was accepted by either 
filter, it was screened for technical feasibility. If the 
task was not rejected it was listed on the Preliminary 
Candidate Task List shown in Table 12. 
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Table 12. PRELIMINARY TASK LIST 



niSSION Ot€— AIR BEFUEUW 

Task" 

Task Br I»f Nueber 

Takeoff 

REVIEWS TAKEOFF OCCKLIST I 

RELEASES BRAKES 2 

SET ENGINE STALL PREVENTION SWITCH TO 3 

CLIMATIC 

CALLS -70 WOTS, NOW" 4 

CALLS -COMMITTED - 4 

SELECTS FLIGHT OIPECTOB FOC HEADING S 

•UATEP-OUT EPR" 4 

CALLS AFTER TRKEOFF-CL IMB CHECKLIST 6 

SELECTS FLIGHT OIPECTOP FOP NAV 5 

guidance 

ENGAGES FU TOP I LOT TO CAPTURE OESIPEO 7 

HEROINS 

PESETS ALTIMETERS 8 

SET THRUST GATE 9 

flir Ref ue I ing 

■LEWI AT FL290- 4 

•BEGIN VOUP TURN* 4 

•VISUAL CONTACT- 4 

•GET PREP CNTCT CL" 4 

"LOUD ANO CLEAR* 4 

OISENGAGE AP 10 

SET EVS FLIP VIDEO 8 

•SPEEO 270, CLIMBING- 4 

AP TO AR MOOE 10 

WATCH AR LITES 11 

•FINAL CONNECT- 4 

■BRKAURY PRAC- 4 

" BREAKAWAYS* 4 

DUMP AP AR MOOE 13 

•CLEAR TANKER" 4 

-GET POST REF CL" 4 



Recovery 

CALL OESCENT CL I ST 6 

SET ALT 12 

SET AIRBRAKES 12 

DUMP AP 13 

00 AG CHUTE 14 

CHECK ALL HYDRAULIC IS 

check cros swing crab IS 

CALL CHECKLIST G 



"TASK NUMBER KEY 

Rev i*wf Takeoff Checklist I 
Pel eases Brakes 2 
Set/Se lect/Engage/Oisengage < A/M 2> 3 
Instruction 4 
Set/Se lect/Engege/O 1 sengage CV/TI 3) S 
Checklist 6 
Set/Select/Engage/Oisengage (U/U 3) 7 
Set/S* lect/Engage/Oisengage ( V/M 1 ) 6 
Set/Select/Engage/Oisengage (V/M 2> 9 
AP/AR (V/M 3) 10 
AP/AR (V/V 2) 11 
Set/Select/Engage/Engage (A/M 1 ) 12 
AP/AR (V/M 1> 13 
Dreg Chute 14 
Checks IS 
Set/^elect/Engage/Oisengage (V/M 3) 16 
Set/Select/Engage/Oisengage (V/V 2) 17 



MISSION TWO — LOWLEVEL 

Task" 

Task Brief Nueber 

Takeoff 

REVIEWS TAKEOFF CHECKLIST I 

RELEASES BRAKES 2 

SET ENCIh€ STALL PREVENTION SWITCH TO 3 

CLIMATIC 

CALLS *70 KNOTS, NOW- 4 

CALLS -COMMITTEO- 4 

SELECTS FLIGHT OIPECTOP FOR HEROINS 5 

•WATER-OUT EPR* 4 

CALLS AFTER TAKEOFF-CLIMB CHECKLIST 6 

SELECTS FLIGHT DIRECTOR FOR NRV S 

GUIDANCE 

ENGAGES AUTOPILOT TO CAPTURE OESIPEO 7 

HEADING 

RESETS ALTIMETERS 8 

SET THRUST GATE 9 

Low I eve I 

■GET LL OESC CL* 4 

TURN ON EVS TA VIO 8 

RAO ALT TO 000 FT 8 

SET ALTS 8 

DUMP RP 13 

AP TO LL 10 

CHK ALTS 15 

SET ALTS 8 

CHECK SEAR, FLAPS UP 15 

•GET TA CHCKLST UP* 4 

. SET ALT HOLD AP 16 

CHK ALTS IS 

■YOUR AIRPLANE" 4 

COMPARE STAB MOOES IS 

SET STAB REF SEL. 5 

PROFILE SET B 

RERO RAO ALT. 15 

SET CL PLANE CAL UAL 8 

SET ALTS 8 

SET STAS REF SEL 9 

SET CL PLANE TO ALT 5 

SET RAOALT CURSOR 8 

•I HAVE THE AC" 4 

TRSRNS BOMB REL LITES 17 

Recovery 

CALL DESCENT CL I ST 6 

SET ALT 12 

SET AIRBRAKES 12 

DUMP AP 13 

DRAG CHUTE 14 

CHECK ALL HYDRAULIC 15 

CHECK CPOSSW1NO CRAB 15 

CALL CHECKLIST 8 
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At this point the tasks were then grouped into types of 
tasks based on whether they accessed the same equipment, had 
the same input/output mode, required the same number of 
steps to complete and were assigned the same anthropometry 
index. Numbers with a corresponding key were used to 
identify like tasks. Each task which occurred more than 
once in either mission was included on the Candidate Task 
List (Table 13). 

Table 13. CANDIDATE TASK LIST 

1. Verbal Instruction 

2. Set/Select/Engage/Disengage 

3. Checklist 

4. Set/Select/Engage/Disengage 

5. Set/Se lect/Engage/Disengage 

6. Adjust AP/AR (V/M 3) 

7. Set/Select/Engage/Disengage 

8. Adjust AP/AR (V/M 1) 

9 . Checks 

B. MODULE TWO 

This module requires that pilots be surveyed and a Pilot 
Utility Index (PUI) be assigned to each of the tasks listed 
in Table 13. For the purposes of this example, it was 
assumed that the pilots had been questioned and the PUI(s) 
listed in Table 14 obtained. 



(V/M 3) 3 

(V/M 1) 
(V/M 2) 

(A/M) 



a 

The information in 
mode and anthropometry 
information is provided 
i ncluded . 



parentheses indicate the input/output 
index of the task listed. This 
when tasks with identical titles are 
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Table 14. PILOT UTILITY INDEX 





Task 




PUI 


1 . 


Verbal Instruction 


(V/M 3/ 0 


4.2 


2. 


Set/Select/Engage/Disengage 


5.2 


3. 


Checklist 




3.5 


4. 


Set /Select /Engage/Disengage 


(V/M 1) 


4.8 


5. 


Set/Select/Engage/Disengage 


(V/M 2) 


4.8 


6 . 


Adjust AP/AR (V/M 3) 




5.5 


7. 


Set/Select/Engage/Disengage 


(A/M) 


4.8 


8. 


Adjust AP/AR (V/M 1) 




5.0 


9. 


Checks 




5.5 



C. MODULE THREE 

Module Three contains the performance specifications 
developed for use in completing the optimization in module 
four. A conversion factor was obtained for each task listed 
on Table 13 based on the number of steps the each task 
requires . 

D. MODULE FOUR 

The optimization problem for this example has the same 
formulation as described in module four of the model 
description contained in Chapter II. The only exception is 
that the type of task indicator is not necessary since no 
single tasks remained for consideration after the filters 
were applied. 

As mentioned in Chapter II, the LINDO software package 
by LINDO, Inc was used to solve this problem. Figure 8 



^®The information in 
mode and anthropometry 
information is provided 
included . 



parentheses indicate the input/output 
index of the task listed. This 
when tasks with identical titles are 
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contains the LINDO problem formulation. For each mission, C 
(the maximum total additional time allowed for voice 
processing), was calculated by deciding the total amount of 
time which could be spent in voice information processing 
channels during a mission and subtracting from it the time 
currently consumed in these channels by non-candidate tasks. 
The time was calculated by using the conversion factor 
calculated above along with the reaction times contained in 
Table 3. The utility value Tj^was calculated by summing 
the reaction time savings (conversion factor x appropriate 
Table 5 entries x number of times tasks performed during the 
mission), pilot utility index, and the anthropometry index. 

Figure 9 contains the solution produced by the LINDO 
package. Tasks two through eight were selected to be 
converted to the audio/vocal mode while task nine was 
selected for conversion to the audio/manual mode. Since 
task one contains a fractional value, it is dropped from 
conversion consideration. This combination of tasks are now 
used by the engineer to begin design and testing. 
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MAX -26.89 Xlll * A. 2 Xll2 ♦ 10.5 X113 ♦ 61.6 X1H ♦ 6.2 X211 ♦ 12.6-1 X212 
♦ 19.53 X213 ♦ 27.16 X21 4 - 3.16 X311 ♦ 3.5 X312 ♦ 4.85 X313 ♦ 15.8 X314 

♦ 5.8 X 4 1 1 ♦ 10.24 X412 ♦ 13.14 X413 ♦ 18.44 X4l4 ♦ 6.8 X511 ♦ 9.02 X512 

♦ 10.47 X513 ♦ 12.8 X514 ♦ 8.5 X611 ♦ 12.94 X612 ♦ 15.84 X613 ♦ 21.14 X614 
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Figure 8. 
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Figure 9. Lindo Solution 
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